The CIFAR-10 dataset (Canadian Institute For Advanced Research) is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10 different classes. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class.
The CIFAR-10 dataset
https://www.cs.toronto.edu/~kriz/cifar.html
https://www.cs.toronto.edu/~kriz/cifar.html
Import packages needed¶
In [1]:
import os
import time
import numpy as np
import pandas as pd
import seaborn as sns
from packaging import version
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import accuracy_score
from sklearn.metrics import mean_squared_error as MSE
from sklearn.model_selection import train_test_split
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
import matplotlib as mpl
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import models, layers
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPool2D, BatchNormalization, Dropout, Flatten, Input, Dense
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping
from tensorflow.keras.preprocessing import image
from tensorflow.keras.utils import to_categorical
In [2]:
%matplotlib inline
np.set_printoptions(precision=3, suppress=True)
Verify TensorFlow Version and Keras Version¶
In [3]:
print("This notebook requires TensorFlow 2.0 or above")
print("TensorFlow version: ", tf.__version__)
assert version.parse(tf.__version__).release[0] >=2
In [4]:
print("Keras version: ", keras.__version__)
Mount Google Drive to Colab Environment¶
In [5]:
from google.colab import drive
drive.mount('/content/drive')
os.chdir('/content/drive/My Drive/Colab Notebooks/MSDS458/Assignment 2/Models/')
Functions for Research Assignment¶
In [6]:
def plot_confusion_matrix(conf_mx):
fig, ax = plt.subplots(figsize=(8,8))
sns.heatmap(conf_mx, annot=True, fmt='.2f', cbar=False, ax=ax, cmap=plt.cm.gray)
plt.ylabel('true label')
plt.xlabel('predicted label')
Loading cifar10 Dataset¶
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
In [7]:
(train_images, train_labels), (test_images, test_labels) = keras.datasets.cifar10.load_data()
- Tuple of Numpy arrays: (x_train, y_train), (x_test, y_test).
- x_train, x_test: uint8 arrays of color image data with shapes (num_samples, 32, 32).
- y_train, y_test: uint8 arrays of digit labels (integers in range 0-9)
EDA Training and Test Datasets¶
- Imported 50000 examples for training and 10000 examples for test
- Imported 50000 labels for training and 10000 labels for test
In [8]:
print('train_images:\t{}'.format(train_images.shape))
print('train_labels:\t{}'.format(train_labels.shape))
print('test_images:\t\t{}'.format(test_images.shape))
print('test_labels:\t\t{}'.format(test_labels.shape))
Review labels for training dataset¶
In [9]:
print("First ten labels training dataset:\n {}\n".format(train_labels[0:10]))
print("This output the numeric label, need to convert to item description")
Plot Examples¶
In [10]:
def get_three_classes(x, y):
def indices_of(class_id):
indices, _ = np.where(y == float(class_id))
return indices
indices = np.concatenate([indices_of(0), indices_of(1), indices_of(2)], axis=0)
x = x[indices]
y = y[indices]
count = x.shape[0]
indices = np.random.choice(range(count), count, replace=False)
x = x[indices]
y = y[indices]
y = tf.keras.utils.to_categorical(y)
return x, y
In [11]:
x_preview, y_preview = get_three_classes(train_images, train_labels)
x_preview, y_preview = get_three_classes(test_images, test_labels)
In [12]:
class_names_preview = ['aeroplane', 'car', 'bird']
def show_random_examples(x, y, p):
indices = np.random.choice(range(x.shape[0]), 10, replace=False)
x = x[indices]
y = y[indices]
p = p[indices]
plt.figure(figsize=(10, 5))
for i in range(10):
plt.subplot(2, 5, i + 1)
plt.imshow(x[i])
plt.xticks([])
plt.yticks([])
col = 'green' if np.argmax(y[i]) == np.argmax(p[i]) else 'red'
plt.xlabel(class_names_preview[np.argmax(p[i])], color=col)
plt.show()
show_random_examples(x_preview, y_preview, y_preview)
Preprocessing Data for Model Development¶
The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents:
| Label | Class_ |
|---|---|
| 0 | airplane |
| 1 | automobile |
| 2 | bird |
| 3 | cat |
| 4 | deer |
| 5 | dog |
| 6 | frog |
| 7 | horse |
| 8 | ship |
| 9 | truck |
In [13]:
class_names = ['airplane'
,'automobile'
,'bird'
,'cat'
,'deer'
,'dog'
,'frog'
,'horse'
,'ship'
,'truck']
Create Validation Data Set¶
In [14]:
train_images_split, valid_images_split, train_labels_split, valid_labels_split = train_test_split(train_images
,train_labels,test_size=.1,random_state=42,shuffle=True)
Confirm Datasets {Train, Validation, Test}¶
In [15]:
print(train_images_split.shape, valid_images_split.shape, test_images.shape)
Rescale Images {Train, Validation, Test}¶
The images are 32x32 NumPy arrays, with pixel values ranging from 0 to 255.
- Each element in each example is a pixel value
- Pixel values range from 0 to 255
- 0 = black
- 255 = white
In [16]:
train_images_norm = train_images_split/255
valid_images_norm = valid_images_split/255
test_images_norm = test_images/255
Experiment 1¶
DNN with 2 layers (no regularization)
Create the Model¶
Build DNN Model¶
In [32]:
model = models.Sequential()
model.add(layers.Flatten(input_shape=(32, 32, 3)))
model.add(layers.Dense(units=108, activation=tf.nn.relu))
model.add(layers.Dense(units=200, activation=tf.nn.relu))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [18]:
model.summary()
In [19]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[19]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [20]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
tf.keras.callbacks.ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint
In [21]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[
tf.keras.callbacks.ModelCheckpoint('/content/drive/My Drive/Colab Notebooks/MSDS458/Assignment 2/Models/model_{val_accuracy:.4f}.h5', save_best_only=True,
save_weights_only=False, monitor='val_accuracy')]
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In order to ensure that this is not a simple "memorization" by the machine, evaluate the performance on the test set.
In [22]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [23]:
history_dict = history.history
history_dict.keys()
Out[23]:
In [24]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [25]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [26]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [27]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [28]:
plot_confusion_matrix(norm_conf_mx)
Predictions¶
Load HDF5 Model Format¶
tf.keras.models.load_model
https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model
https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model
In [29]:
model = tf.keras.models.load_model('/content/drive/My Drive/Colab Notebooks/MSDS458/Assignment 2/Models/model_0.4970.h5')
In [31]:
preds = model.predict(test_images_norm)
preds.shape
Out[31]:
Visualize predictions¶
In [32]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [33]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[33]:
Plot TSNE plot¶
In [34]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-3]
output_layer_activations = activations[-1]
In [35]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [36]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 2¶
DNN with 3 layers (no regularization)
Create the Model¶
Build DNN Model¶
In [37]:
model = models.Sequential()
model.add(layers.Flatten(input_shape=(32, 32, 3)))
model.add(layers.Dense(units=108, activation=tf.nn.relu))
model.add(layers.Dense(units=200, activation=tf.nn.relu))
model.add(layers.Dense(units=282, activation=tf.nn.relu))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [38]:
model.summary()
In [39]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[39]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [40]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [41]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [42]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [43]:
history_dict = history.history
history_dict.keys()
Out[43]:
In [44]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [45]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [46]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [47]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [48]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [49]:
preds = model.predict(test_images_norm)
preds.shape
Out[49]:
In [50]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [51]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[51]:
Plot TSNE plot¶
In [52]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-4]
output_layer_activations = activations[-1]
In [53]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [54]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 3¶
CNN with 2 convolution/max pooling layers (no regularization)
Create the Model¶
Build CNN Model¶
In [55]:
model = models.Sequential()
model.add(layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu,input_shape=(32, 32, 3)))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Conv2D(filters=108, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Flatten())
model.add(layers.Dense(units=210, activation=tf.nn.relu))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [56]:
model.summary()
In [57]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[57]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [58]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [59]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [60]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [61]:
history_dict = history.history
history_dict.keys()
Out[61]:
In [62]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [63]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [64]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [65]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [66]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [67]:
preds = model.predict(test_images_norm)
preds.shape
Out[67]:
In [68]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [69]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[69]:
Plot feature map¶
In [70]:
(_,_), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
img = test_images[2004]
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
class_names = ['airplane'
,'automobile'
,'bird'
,'cat'
,'deer'
,'dog'
,'frog'
,'horse'
,'ship'
,'truck']
plt.imshow(img, cmap='viridis')
plt.axis('off')
plt.show()
In [71]:
# Extracts the outputs of the top 8 layers:
layer_outputs = [layer.output for layer in model.layers[:8]]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
In [72]:
activations = activation_model.predict(img_tensor)
len(activations)
Out[72]:
In [73]:
layer_names = []
for layer in model.layers:
layer_names.append(layer.name)
layer_names
Out[73]:
In [74]:
# These are the names of the layers, so can have them as part of our plot
layer_names = []
for layer in model.layers[:4]:
layer_names.append(layer.name)
images_per_row = 16
# Now let's display our feature maps
for layer_name, layer_activation in zip(layer_names, activations):
# This is the number of features in the feature map
n_features = layer_activation.shape[-1]
# The feature map has shape (1, size, size, n_features)
size = layer_activation.shape[1]
# We will tile the activation channels in this matrix
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))
# We'll tile each filter into this big horizontal grid
for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
# Post-process the feature to make it visually palatable
channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
# Display the grid
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
plt.show();
Plot TSNE plot¶
In [75]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [76]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [77]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 4¶
CNN with 3 convolution/max pooling layers (no regularization)
Create the Model¶
Build CNN Model¶
In [78]:
model = models.Sequential()
model.add(layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu,input_shape=(32, 32, 3)))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Conv2D(filters=108, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Conv2D(filters=180, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Flatten())
model.add(layers.Dense(units=210, activation=tf.nn.relu))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [79]:
model.summary()
In [80]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[80]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [81]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [82]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [83]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [84]:
history_dict = history.history
history_dict.keys()
Out[84]:
In [85]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [86]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [87]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [88]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [89]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [90]:
preds = model.predict(test_images_norm)
preds.shape
Out[90]:
In [91]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [92]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[92]:
Plot feature map¶
In [93]:
(_,_), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
img = test_images[2004]
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
class_names = ['airplane'
,'automobile'
,'bird'
,'cat'
,'deer'
,'dog'
,'frog'
,'horse'
,'ship'
,'truck']
plt.imshow(img, cmap='viridis')
plt.axis('off')
plt.show()
In [94]:
# Extracts the outputs of the top 8 layers:
layer_outputs = [layer.output for layer in model.layers[:8]]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
In [95]:
activations = activation_model.predict(img_tensor)
len(activations)
Out[95]:
In [96]:
layer_names = []
for layer in model.layers:
layer_names.append(layer.name)
layer_names
Out[96]:
In [97]:
# These are the names of the layers, so can have them as part of our plot
layer_names = []
for layer in model.layers[:6]:
layer_names.append(layer.name)
images_per_row = 16
# Now let's display our feature maps
for layer_name, layer_activation in zip(layer_names, activations):
# This is the number of features in the feature map
n_features = layer_activation.shape[-1]
# The feature map has shape (1, size, size, n_features)
size = layer_activation.shape[1]
# We will tile the activation channels in this matrix
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))
# We'll tile each filter into this big horizontal grid
for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
# Post-process the feature to make it visually palatable
channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
# Display the grid
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
plt.show();
Plot TSNE plot¶
In [98]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [99]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [100]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 5¶
DNN with 2 layers with regularization (batch normalization, dropout, early stopping)
Create the Model¶
Build DNN Model¶
In [101]:
model = models.Sequential()
model.add(layers.Flatten(input_shape=(32, 32, 3)))
model.add(layers.BatchNormalization())
model.add(layers.Dense(units=108, activation=tf.nn.relu))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(units=200, activation=tf.nn.relu))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [102]:
model.summary()
In [103]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[103]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [104]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
tf.keras.callbacks.ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint
In [105]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In order to ensure that this is not a simple "memorization" by the machine, evaluate the performance on the test set.
In [106]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [107]:
history_dict = history.history
history_dict.keys()
Out[107]:
In [108]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [109]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [110]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [111]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [112]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [113]:
preds = model.predict(test_images_norm)
preds.shape
Out[113]:
In [114]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [115]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[115]:
Plot TSNE plot¶
In [116]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-4]
output_layer_activations = activations[-1]
In [117]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [118]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 6¶
DNN with 3 layers with regularization (batch normalization, L2 regulariazation, dropout, early stopping)
Create the Model¶
Build DNN Model¶
In [119]:
model = models.Sequential()
model.add(layers.Flatten(input_shape=(32, 32, 3)))
model.add(layers.BatchNormalization())
model.add(layers.Dense(units=108, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(units=200, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(units=282, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [120]:
model.summary()
In [121]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[121]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [122]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [123]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [124]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [125]:
history_dict = history.history
history_dict.keys()
Out[125]:
In [126]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [127]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [128]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [129]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [130]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [131]:
preds = model.predict(test_images_norm)
preds.shape
Out[131]:
In [132]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [133]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[133]:
Plot TSNE plot¶
In [134]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-4]
output_layer_activations = activations[-1]
In [135]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [136]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 7¶
CNN with 2 convolution/max pooling layers with regularization (L2 regularization, dropout, early stopping)
Create the Model¶
Build CNN Model¶
In [20]:
model = models.Sequential()
model.add(layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu, input_shape=(32, 32, 3)))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=108, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.Dense(units=210, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [21]:
model.summary()
In [22]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[22]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [23]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [24]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [25]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [26]:
history_dict = history.history
history_dict.keys()
Out[26]:
In [27]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [28]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [29]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [30]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [31]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [32]:
preds = model.predict(test_images_norm)
preds.shape
Out[32]:
In [33]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [34]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[34]:
Plot feature map¶
In [35]:
(_,_), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
img = test_images[2004]
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
class_names = ['airplane'
,'automobile'
,'bird'
,'cat'
,'deer'
,'dog'
,'frog'
,'horse'
,'ship'
,'truck']
plt.imshow(img, cmap='viridis')
plt.axis('off')
plt.show()
In [36]:
# Extracts the outputs of the top 8 layers:
layer_outputs = [layer.output for layer in model.layers[:8]]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
In [37]:
activations = activation_model.predict(img_tensor)
len(activations)
Out[37]:
In [38]:
layer_names = []
for layer in model.layers:
layer_names.append(layer.name)
layer_names
Out[38]:
In [39]:
# These are the names of the layers, so can have them as part of our plot
layer_names = []
for layer in model.layers[:6]:
layer_names.append(layer.name)
images_per_row = 16
# Now let's display our feature maps
for layer_name, layer_activation in zip(layer_names, activations):
# This is the number of features in the feature map
n_features = layer_activation.shape[-1]
# The feature map has shape (1, size, size, n_features)
size = layer_activation.shape[1]
# We will tile the activation channels in this matrix
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))
# We'll tile each filter into this big horizontal grid
for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
# Post-process the feature to make it visually palatable
channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
# Display the grid
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
plt.show();
Plot TSNE plot¶
In [40]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [41]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [42]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 8¶
CNN with 3 convolution/max pooling layers with regularization (L2 regularization, dropout, early stopping)
Create the Model¶
Build CNN Model¶
In [35]:
model = models.Sequential()
model.add(layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu,input_shape=(32, 32, 3)))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=108, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=180, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.Dense(units=210, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [36]:
model.summary()
In [37]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[37]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [38]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [39]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [40]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [41]:
history_dict = history.history
history_dict.keys()
Out[41]:
In [42]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [43]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [44]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [45]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [46]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [47]:
preds = model.predict(test_images_norm)
preds.shape
Out[47]:
In [48]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [49]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[49]:
Plot feature map¶
In [50]:
(_,_), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
img = test_images[2004]
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
class_names = ['airplane'
,'automobile'
,'bird'
,'cat'
,'deer'
,'dog'
,'frog'
,'horse'
,'ship'
,'truck']
plt.imshow(img, cmap='viridis')
plt.axis('off')
plt.show()
In [51]:
# Extracts the outputs of the top 8 layers:
layer_outputs = [layer.output for layer in model.layers[:8]]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
In [52]:
activations = activation_model.predict(img_tensor)
len(activations)
Out[52]:
In [53]:
layer_names = []
for layer in model.layers:
layer_names.append(layer.name)
layer_names
Out[53]:
In [54]:
# These are the names of the layers, so can have them as part of our plot
layer_names = []
for layer in model.layers[:8]:
layer_names.append(layer.name)
images_per_row = 16
# Now let's display our feature maps
for layer_name, layer_activation in zip(layer_names, activations):
# This is the number of features in the feature map
n_features = layer_activation.shape[-1]
# The feature map has shape (1, size, size, n_features)
size = layer_activation.shape[1]
# We will tile the activation channels in this matrix
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))
# We'll tile each filter into this big horizontal grid
for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
# Post-process the feature to make it visually palatable
channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
# Display the grid
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
plt.show();
Plot TSNE plot¶
In [55]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [56]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [57]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 9¶
CNN with 3 convolution/max pooling layers with regularization (L2 regularization, dropout, early stopping) and increased complexity using a larger number of filters, nodes, and epochs than Experiment 8.
Create the Model¶
Build CNN Model¶
In [10]:
model = models.Sequential()
model.add(layers.Conv2D(filters=128, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu,input_shape=(32, 32, 3)))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=216, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=360, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.Dense(units=420, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [11]:
model.summary()
In [12]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[12]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [13]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [14]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=50
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [15]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [16]:
history_dict = history.history
history_dict.keys()
Out[16]:
In [17]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [18]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [19]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [20]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [21]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [22]:
preds = model.predict(test_images_norm)
preds.shape
Out[22]:
In [23]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [24]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[24]:
Plot TSNE plot¶
In [25]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [26]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [27]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
Experiment 10¶
CNN with 4 convolution and 2 max pooling layers with regularization (batch normalization, L2 regulariazation, dropout, early stopping)
Create the Model¶
Build CNN Model¶
In [43]:
model = models.Sequential()
model.add(layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu,input_shape=(32, 32, 3)))
model.add(layers.Conv2D(filters=108, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D((2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Conv2D(filters=180, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.Conv2D(filters=210, kernel_size=(3, 3), strides=(1, 1), activation=tf.nn.relu))
model.add(layers.MaxPool2D(pool_size=(2, 2),strides=2))
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.BatchNormalization())
model.add(layers.Dense(units=260, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.L2(0.001)))
model.add(layers.Dense(units=10, activation=tf.nn.softmax, name="output_layer"))
In [44]:
model.summary()
In [45]:
keras.utils.plot_model(model, "CIFAR10.png", show_shapes=True)
Out[45]:
Compiling the model¶
tf.keras.losses.SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
In [46]:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
Training the model¶
In [47]:
start = time.time()
history = model.fit(train_images_norm
,train_labels_split
,epochs=30
,batch_size=500
,validation_data=(valid_images_norm, valid_labels_split)
,callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=3)],
)
print("Total time: ", time.time() - start, "seconds")
Evaluate the model¶
In [48]:
loss, accuracy = model.evaluate(test_images_norm, test_labels)
print('test set accuracy: ', accuracy)
Plotting Performance Metrics¶
Matplotlib is used to create 2 plots -- displaying the training and validation loss (resp. accuracy) for each (training) epoch side by side.
In [49]:
history_dict = history.history
history_dict.keys()
Out[49]:
In [50]:
losses = history.history['loss']
accs = history.history['accuracy']
val_losses = history.history['val_loss']
val_accs = history.history['val_accuracy']
epochs = len(losses)
In [51]:
plt.figure(figsize=(16, 4))
for i, metrics in enumerate(zip([losses, accs], [val_losses, val_accs], ['Loss', 'Accuracy'])):
plt.subplot(1, 2, i + 1)
plt.plot(range(epochs), metrics[0], label='Training {}'.format(metrics[2]))
plt.plot(range(epochs), metrics[1], label='Validation {}'.format(metrics[2]))
plt.legend()
plt.show()
Confusion matrices¶
Using sklearn.metrics, visualize the confusion matrix.
In [52]:
pred1= model.predict(test_images_norm)
pred1 = np.argmax(pred1, axis=1)
In [53]:
conf_mx = confusion_matrix(test_labels, pred1)
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = np.round((conf_mx / row_sums), 2)
In [54]:
plot_confusion_matrix(norm_conf_mx)
Visualize predictions¶
In [55]:
preds = model.predict(test_images_norm)
preds.shape
Out[55]:
In [56]:
cm = sns.light_palette((260, 75, 60), input="husl", as_cmap=True)
In [57]:
df = pd.DataFrame(preds[0:20], columns = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'])
df.style.format("{:.2%}").background_gradient(cmap=cm)
Out[57]:
Plot TSNE plot¶
In [58]:
# Extracts the outputs of all layers:
layer_outputs = [layer.output for layer in model.layers]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
# Get activation values for the last dense layer
activations = activation_model.predict(valid_images_norm[:5000])
dense_layer_activations = activations[-2]
output_layer_activations = activations[-1]
In [59]:
# Reduce the dimension using T-SNE to visualize i n a scatterplot
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
tsne_results = tsne.fit_transform(dense_layer_activations)
# Scaling
tsne_results = (tsne_results - tsne_results.min()) / (tsne_results.max() - tsne_results.min())
In [60]:
cmap = plt.cm.tab10
plt.figure(figsize=(16,10))
scatter = plt.scatter(tsne_results[:,0],tsne_results[:,1], c=valid_labels_split[:5000], s=10, cmap=cmap)
plt.legend(handles=scatter.legend_elements()[0], labels=class_names)
image_positions = np.array([[1., 1.]])
for index, position in enumerate(tsne_results):
dist = np.sum((position - image_positions) ** 2, axis=1)
if np.min(dist) > 0.02: # if far enough from other images
image_positions = np.r_[image_positions, [position]]
imagebox = mpl.offsetbox.AnnotationBbox(
mpl.offsetbox.OffsetImage(train_images[index], cmap="binary"),
position, bboxprops={"lw": 1})
plt.gca().add_artist(imagebox)
plt.axis("off")
plt.show()
